Job Description
We are looking for a talented Data Engineer to be part of our Data team. You will work alongside a growing multidisciplinary team of talented research scientists and machine learning engineers to improve and scale the efficiency within our models. You will contribute in building the data infrastructure that manages our data and powers the training of all Stability AI models. The position is based in Japan and is a hybrid work model with at least one day (recommended) in the office in Tokyo per week.
Responsibilities
- Clean, normalize, and preprocess data in a scalable, parallelizable way to prepare it for ingestion into our machine learning model training pipelines while ensuring data quality
- Design, implement, and maintain scalable data infrastructure for generative AI
- Develop tools to search and serve the data at scale
- Collaborate with cross-functional teams to understand and meet their data requirements
- Develop and manage data processing pipelines to support machine learning teams
- Maintain and improve data quality and integrity across various databases and data stores
- Manage and organize large-scale unstructured data, including image, text, audio, video and 3D
Qualifications
- Proven background within large-scale distributed workloads
- Experience with large-scale data loading for machine learning training runs
- Experience with cloud storage and file systems. AWS (S3) is strongly preferred, but open to other cloud platforms
- Experience with Python
- Experience with large-scale data processing and software development for unstructured data.
- Expertise in database, data lake, and data warehouse technologies (Redshift, BigQuery, Snowflake).
- Experience working with machine learning projects and ideally some deep learning / computer vision knowledge
- Good teamwork and communication skills based on experience working with a distributed international team with timezone and cultural differences
- Excellent communication skills to effectively collaborate with users, solve issues, and provide guidance
- Attention to detail and the ability to document processes and solutions effectively
About the Company
Stability AI is a community and mission driven, open artificial intelligence company that cares deeply about real-world implications and applications. Our most considerable advances grow from our diversity in working across multiple teams and disciplines. We are unafraid to go against established norms and explore creativity. We are motivated to generate breakthrough ideas and convert them into tangible solutions. Our vibrant communities consist of experts, leaders and partners across the globe who are developing cutting-edge open AI models for Image, Language, Audio, Video, 3D and Biology.
Equal Employment Opportunity
We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses.